\name{qapply}
\alias{qapply}

\title{
Parallel lapply using grid engine
}
\description{
The qapply function will attempt to replicate the behavior of lapply using grid
engine, applying a function to each element of the first argument (\code{X})
and returning a list.
}
\usage{
qapply(X, FUN, fargs, commonData = NULL, workDir = getwd(), nCores =
as.numeric(.getResources()), nPerPacket = NULL, tag =
deparse(substitute(X)), internalize=TRUE, clearWd=TRUE, parSeed=NULL)
}

\arguments{
  \item{X}{
    An array or list of values to be evaluated by \code{FUN}
}
  \item{FUN}{
    A function to be applied to \code{X}
}
  \item{fargs}{
    A list of (named) arguments for \code{FUN}
}
  \item{commonData}{
    A list of (named) objects that will be visible to each child process
}
  \item{workDir}{
    Where the worker processes will be spawned, and where the "out",
  "err", and "log" temporary files will be created
}
  \item{nPerPacket}{
    A vector of integers whose sum is \code{length(X)}.  Packet \code{i}
  contains \code{nPerPacket[i]} elements of \code{X}.
}
  \item{nCores}{
    The number of computational cores to be used for the \code{length(X)} individual
    jobs.  When \code{nPerPacket} is unspecified, \code{qapply} will
  build \code{nCores} packets of jobs to reduce overhead tied to configuring the worker processes.
    The default value of \code{nCores} is to take the number available to the grid at the
  time of \code{qapply}'s execution.  
}
  \item{tag}{
    A label for the job and temporary files related to the job.  The
  default is to take the name of \code{X}.
}
\item{internalize}{
  When TRUE, \code{qapply} will return a list containing the output of
  \code{FUN} for each element of \code{X}.  When FALSE, \code{qapply}
  acts simply to facilitate submission of jobs to the queue.  See \code{qsub}.
}
\item{parSeed}{
  A vector of six integers which is passed to the \code{rlecuyer}
  package for generating parallel random number streams.  If left blank,
  the default is \code{rep(1111, 6)}.  See \code{qseed}
}
}
\details{
   The \code{qapply} function utilizes the wrapper \code{\link{qsub}} to
   perform an lapply.  This is done by instructing worker processes to
  evaluate the function on subsets of \code{X} called packets.  Each
  packet contains a number of jobs, where each job is a single
  application of \code{FUN} to an element of \code{X}. 
   
   The finest granularity is achieved by setting
   \code{nPerPacket=rep(1, length(X))}.  While useful on expandable
   clouds (i.e., nCores is not fixed), this behavior explicitly implies
   that \code{length(X)} R processes will be loaded and configured.  The
   time it takes to set up each of these processes depends upon the size
   of the \code{commonData} and \code{fargs} objects as these must be loaded in the
   new environment.

   The library path of the parent environment and all loaded libraries will be preserved in child processes.  At this time, however, the same can not be said for "options". 

   Upon execution \code{qapply} assembles "packets" of jobs to be
   evaluated on a single core, writes out the
   common environment as an R object, and creates the "out", "log", and
   "err" subdirectories of \code{workDir}.  The "log" and "err"
   subdirectories can be helpful in debugging and will remain intact
   unless clearWd is set to TRUE.  By default (with \code{internalize=TRUE}) the R process will wait for
   all jobs to complete (see \code{\link{qwait}}).  When complete, the
   temporary output files in \code{file.path(workDir, "out")} will be
   aggregated and returned as a list using \code{\link{qinternalize}}.  
}
\value{
  When internalize=TRUE, A list resulting from the application of \code{FUN} to \code{X}.
}
\references{
%% ~put references to the literature/web site here ~
}
\author{
Dan Polhamus
}
\seealso{
\code{\link{qsub}}, \code{\link{rlecuyer}}, \code{\link{qinternalize}}, \code{\link{qstat}},
\code{\link{qseed}}, \code{\link{qwait}}
}
\examples{
whatLetter <- function(letter){
  number <- which(letter == letters)
  ## Prefix is some data defined in the parent environment
  paste(prefix, letter, "is letter number", number)
}

prefix <- "This letter:"
system.time(parallel <- qapply(letters, whatLetter, commonData=list(
prefix=prefix )))

## This is much faster without using grid engine, as will always be the
## case when the evaluation of FUN is fast.


## -----------------------------------------------------------------------------

invMatrix <- 

qapply(rep(500, 1000), function(nr){
  solve( matrix(rnorm(nr^2), nrow=nr, ncol=nr) )
}, internalize=FALSE, tag="invertMatrix")

parallel <- qinternalize("out/invertMatrix")
qclearWd()

 
}

% Add one or more standard keywords, see file 'KEYWORDS' in the
% R documentation directory.
\keyword{ parallel }
\keyword{ HPC }% __ONLY ONE__ keyword per line
\keyword{ SGE }
